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Abstract: In this paper we investigate quasi-stationary distribu- 
tions [i of stochastic approximation algorithms with constant step 
size which can be viewed as random perturbations of a time-continuous 
dynamical system. Inspired by ecological models these processes have 
a closed absorbing set corresponding to extinction. Under some large 
deviation assumptions and the existence of an interior attractor for 
the ODE, we show that the weak* limit points of the QSD fj, are 
invariant measures for the ODE with support in the interior attrac- 
tors. 
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1. Introduction. 

One of the most considered issue in theoretical ecology is to find out under 
what kind of conditions one can expect a population of interacting species 
(animals, plants, microorganisms, ...) to survive on the long term with no 
extinctions. When these conditions are met the interacting populations are 
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said to persist or coexist. In the past, differential equations and nonlin- 
ear difference equations have been used to model these phenomena. Famous 
examples are Lotka [14] and Volterra [22] work on competitive and predator- 
prey interactions, Thompson [21], Nicholson and Bailey [16] on host-parasite 
interactions, and Kermack and McKendrick [12] on disease outbreaks. For 
these deterministic models, persistence definitions sometimes vary but most 
authors link persistence with the existence of an attractor bounded away 
from the extinction states, in which case persistence holds over an infinite 
time horizon, see e.g. [20]. In order to refine these models and allow for some 
"roughness" and/or influence of unpredictable outer events, randomness has 
been added to these models, leading to Markov processes models. However, 
extinctions being absorbent states and species dying out with positive prob- 
abilities, the underlying theory of Markov processes shows that, in finite 
time, extinction is inevitable. Yet, in the real world, with large sized pools 
of population, we don't observe that inevitable extinction. This finite extinc- 
tion time may then be very large and the system may remain in some sort 
of "metastable state" bounded away from extinction for a long time. In [10], 
Faure and Schreiber studied this problem for randomly perturbed discrete 
time dynamical systems, showing that, under the appropriate assumptions 
about the random perturbations, there exists a positive attractor (i.e. an at- 
tractor which is bounded away from extinction states) for the unperturbed 
system, which implies two things as the number of individuals or particles 
gets large. First, when they exist, quasi-stationary distributions concentrate 
on the positive attractors of the unperturbed system. Second, the expected 
time to extinction for systems starting according to this quasi-stationary 
distribution grows exponentially with the system size. The aim of this paper 
is to extend their approach to a class of discrete time Markov process, that, 
up to a renormalization of time, can be seen as random perturbations of an 
ordinary differential equation. 

In Section 2 we will introduce our setting and give some examples of systems 
that fall into it. Then, in Section 3 we will show that, under the hypoth- 
esis that the deterministic mean dynamic admits an interior attractor, the 
extinction time grows exponentially with the size of the system and that, 
when the system size goes to infinity, the limit set of the quasi-stationary 
distributions of the processes for the weak* convergence consists of invariant 
measures for the deterministic dynamic. Finally in Section 4 we will study 
the support of these invariant limiting measures and prove that, under some 
additional large deviations hypotheses, their support lies within attractors 
bounded away from the extinction states. To do that we will compare two 
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different notions of chain-recurrence, one given by the large deviations func- 
tional and the other a slight variation on Conley's S-T chain-recurrence. 
Should the reader need some reminders about the objects used in this pa- 
per, he will find some basic properties and some references in the Appendix 
at the end. 

2. Model, notations and hypotheses. 

We denote by A the d-dimensional simplex. 

d 

A = {x G R d ; Vi = 1 • • • d x { ^ k x { = 1} 

i=i 

We let A denote the relative interior of A and 

A N = A n 

A N = An±z d . 

Let F : A — > R d be a locally Lipschitz vector field such that : 

d 

\fxeA ^2Fi(x) = 
i=i 

Unless specified otherwise, the topology considered will be the topology 
induced by the classical R d metric topology on A. Throughout the paper, 
if A is a subset of a metric space (E,d), we will denote by N £ (A) its e- 
neighborhood 

N £ (A) = { x eE; d(x,A) < e}. 

We consider a family of Markov chains (Xn)neN defined on a probability 
space (fi, F, P) taking values in the d-dimensional discrete simplex A^r- 

We denote by F^ the u-algebra generated by {X^,i = l,...,n}. For 
A e F we let F X [A] = F[A\X = x]. 

Throughout the paper the following hypothesis will always be assumed 
to hold. 

Standing Hypothesis 2.1 : 

The Markov process X N has the following properties : 
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(ii) E[U». 1 \F»] = 

(Hi) There exists T ^ such that \\U^\\ ^ vT 

(iv) The boundary of the simplex is an absorbing set: 

(a) for all x G dA F X [X? G dA] = 1 

(b) for all x £ A F x [3n : X% G dA] = 1 
(ty 1^ restricted to A* N is irreducible 

Vx,yeA* N ¥ x [3n : X n = y] > 

anc? aperiodic 

Vx G A^v 5 cd({n ; P x [X n = x] > 0}) = 1 



Example 1 (Guiding Thread): 

Let (Pi,j(aj))i,_jg{i...d} be a family of real- valued continuous functions on A 
such that, for all x G A : 

(1) Vi/j f>i,j(x) = XiXj = 0, 

(2) Pi,i(a;) = 0, 

(3) 0<pij(x)<l, 

(4) x; ^(x) < i. 



Let ) be the random walk on An defined by: 

= Pi,j(x) 



X& 1 = Xj? + ±(e j -e i )\X£ = x 



where (ei)i=\—d is the canonical base of M. d . This type of model often occurs 
in population games. In this setting N represents the size of the population. 
Each individual plays a pure strategy % and X N represents then the vector 
of proportion of players of each strategy. The jump Xj^ +1 = X^ + jj(ej — e%) 
means that an individual switches his strategy from i to j at time k. The 
conditions on the family (pij) mean that : 

• At each time k, it is always possible that a player switches from his 
strategy i to another strategy j that is currently in use in the popula- 
tion. 

• No individual switches to an unused strategy. This makes sense for 
models based on strategy switching from imitations or models arising 
from ecology. 
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We define 

d 

Pi( x ) = ^2pjA x ) 

d 

i=i 

p(x) the vector of coordinates Pi(x) and q{x) the vector of coordinates qi{x). 

In this case F(x) = p(x) — q(x). Hypothesis 2.1(i) comes from the Markov 
property, 2.1(h) and 2.1 (hi) follow easily from the definition of the chain and 
2.1(iv) and 2.1(v) follow from the fact that the functions Pij are positive on 
the relative interior and vanish on the boundary. 

A class of examples that falls in Example 1 setting is given by Imitative 
Protocols games, see e.g. [19]. Consider a population game with d pure 
strategies, we let U[x) = (Ui(x), ■ ■ ■ Ud{x)) denote the vector of payoffs 

d 

when the population is in state x S A and U(x) = ^^XjC/j(x) denotes the 

i=i 

average payoff at population state x. Imitative protocols are of the form 

Pi,j(x) = XiXjrij(U(x),x) 

with some additional assumptions on to ensure that the problem is well- 
posed. Under such a protocol, at each time we pick an individual uniformly 
at random among the population and give him a revision opportunity. The 
opportunity unfolds as such 

• The individual picks an opponent uniformly at random among the 
population (he/she can pick him/her-self) and observes his/her strat- 
egy 

• If the individual plays i and the opponent plays j then the individual 
switches from i to j with probability proportional to r^-. 

We now give some examples of Imitative Protocols 
Example 2 (Pairwise Proportional Imitation): 

After selecting an opponent the agent imitates only if the opponent's payoff 
is higher than his own, doing so with probability proportional to the payoff 
difference. 

Pij(x) = XiXj(Uj(x) - Ui(x)) + 
where (y) + stands for max(y,0). 
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The mean dynamic generated by this protocol is 

d d 



d 



= j2 x i x i c^o*) - u ^ x )) + - ^Ec^o*) - Ui ^ + 

i=i 3=1 

d 

= Xi^xjiUiix) - Uj(x)) 

3=1 

= x i (U l {x)-U{x)) 

We then get the well known replicator dynamic, a dynamic extensively 
studied in ecology and evolutionary game theory, see e.g. [11] 

Example 3 (Aspiration and Random Imitation): 

A particular case of the former example is the aspiration and random imi- 
tation model, see e.g. [3], [4], [6] and [5]. At each time we pick an individual 
at random in the population and look at his/her "satisfaction", a payoff-like 
function Ui(x) where i is the type of the drawn individual. If this satisfac- 
tion is lower than a certain aspiration level then switch to another type 
chosen at random in the population, otherwise stay at current type. The 
morality of this model is that, if your type isn't performant enough for 
your tastes then switch to another type. The aspiration levels are indepen- 
dent random variables uniformly distributed on intervals [di(x), bi{x)\ with 
a,i(x) ^ Ui(x) ^ bi(x). This model gives us 

_ bj(x) - Uj(x) 

Pi,j\ x ) — XiXj . . . . 

J bi{x) -ai(x) 

If we assume that the aspiration level bounds are not type-dependant, mean- 
ing a,i(x) = d{x) and bi{x) = b(x) for all i then we get a mean field given 

by 

. _ uj{x) - yi'' . »,(./•) 

b(x) — d(x) 

which is a replicator dynamic with fitness functions fi{x) = b r^!fJ( x \ 

Alternatively we can assume that the aspiration levels follow the type 
payoff by the relation bi(x) = f3iUi(x) and di(x) = QjMj(x) with ctj < 1 < 
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In this case we get a dynamic 




where v« = -a 2 — . 

Again this is a replicator dynamics with fitness function fi(x) = —V{. 

Example 4 (Imitation Driven by Dissatisfaction): 

In this protocol, when a i player receives a revision opportunity, he opts to 
switch strategies with a probability that that is linearly decreasing in his 
current payoff. Should he decide to change, then he will imitate a randomly 
selected opponent. This protocol also falls under the former example of as- 
piration games with constant aspiration levels a,(x) = A, bi{x) = B. This 
gives the following dynamic 

x i = ^- J (F i (x)-F(x)) 

Again we recognize a replicator dynamic. 
For other examples see e.g. [19]. 

3. Convergence of QSD and absorption time. 

We denote by {<£>t} the flow induced by F. In order to compare the tra- 
jectory of (ft with those of (X^) it's convenient to introduce the continuous 
process defined by 

X N (k/N) =Xf? V/c G N 

and extended on every interval [k/N,(k + l)/N] by piecewise linear interpo- 
lation. 
Let 

^(r)= o rna^||X Ar (t)-^(X ^)|| 

be the variable measuring the distance between the trajectories t i— > X (t) 
and t H> ipt(Xff). 

We recall this convergence theorem of Benai'm and Weibull. 

Theorem 3.1 : 

For every T > 0, there exists c > (depending only on F,T and T) such 
that, for every e > 0, and for N large enough : 

P[D N (T) ^e}^ 2de~ £2cN 
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For a detailed proof of this result, see [3]. 
We define to be the absorption time. 

Tj? = inf{n > 0; 6 dA} 



Hypothesis 2.1(iv) implies that, whatever the initial state is, the process will 
almost surely be absorbed, i.e. 



A probability measure p on the discrete relative interior of the simplex 
A* N is said to be a quasi- stationary distribution, thereafter referred as QSD, 
if and only if, for every Borelian set A C and every n > 0, 



We remark that, in this case, p is a fixed point for the conditional evolution 



The following proposition is a classic QSD result and follows easily from 
the Perron-Frobenius theorem. 

Proposition 3.2 : 

For every N, there exists an unique quasi-stationary distribution p N ob- 
tained as the only left eigenvector p N of the transition matrix of the Markov 
chain restricted to A* N verifying 



y x £ a 



P X [T» < oo] = 1. 



P„[JC eA\T N >n]=p(A). 



u^F u [X% G .\T >n] 



V*€ {!,•••(*} Mf>0 ; 5>f = l 



The corresponding eigenvalue < pn = e N < 1 is such that 



F^[TF>n]=e- e » 



Hence, starting from p , the expectation of Tq is 




>N 



1 







1 - PN 



Proof : 

For a detailed proof see e.g. [15]. 
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A set A C A is called an attractor for the flow {(ft} if 

(i) A is compact and invariant, i.e. for every tGl <Pt(A) = A. 

(ii) There exists a neighborhood U of A, called a fundamental neighbor- 
hood, such that 

lim dUpAx), A) = 

t— >oo 

uniformly in x in U. 
Theorem 3.3 : 

Starting from fi N , the law of the absorption time and its expectation are 
given by Proposition 3.2. If we further assume that the flow {<pt} admits an 
attractor A C A, then, there exists 7 > such that the following estimate 
holds : 



0^1 — pn O 



e -fN 



N 

Thus, there exists a constant C > such that 

E^n[T ] > CNe lN 

Proof : 

Let V C A and let k G N. By the QSD property we have: 
P k N^ N (V)= £ p k N (x,V)^ N (x) 

> £ p k N (x,V)^ N (x) 
x£VnA N 

> inf 

xevnA N 



Thus 



zeFnAjv 



Let C/ C A be a fundamental neighborhood of the attractor A. We know 
that d(tpt(x), A) converges uniformly to over U. Hence 

Ve>0 3T(e)>0 Vi ^ T(e) G U d(p t (x) , A) < e . 

Let a = d(A, U c ), e < a, T = T(e) and 8 <a — e. 
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For all x £ U n A N 

p [ £ T] (x,U c ) ^V x [d(X[NT\,A) > a] 

^F x [d(X^ T] ,ip T (x)) >a-s] 



Then 



Therefore 



^ F x [d(X[NT\,Xi ' ) + d(X Tt <pr(x)) >a-e 

-8 2 cN 



D N {T)>a-e- (||F|| + y/f) NT 



< 2de 



for N large enough (see Theorem 3.1) 



a;G(7nAjv 

^1— max p^ T \x,U c 
x£UnA N N 



> 1 - 2de 



-<5 2 cAT 



1 - pat < 1 - 1 - 2de 



-<5 2 c7V\ Pv? 



1 - 2de 



S 2 cN\ Wt] = p^rjlog {l-2de- s2 " N ) 



In conclusion we have 



-i 2dc~ s cN 
e [JVT] Zae 

. 2de" 



[NT] 



+ o{e- &2cN )) 
( 2de 

+ o 



[NT] 



< 1 - pat < O 



N 



3.2. Convergence of the QSD to an invariant measure. 



A probability measure /j on A is called an invariant measure for the flow 
{<p t } if, for all £ G R and all borelian set A £ B(A), //(^T 1 ^)) = //(A). 
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Theorem 3.4 : 

We suppose that the flow {<£t} admits an attractor Ac A, Then the set of 
limit points of{fi N } for the weak* topology is a subset of the set of invariant 
measures for the flow {ft}- 
Proof : 

Let / be a Lipscbitz function from A to 1 with constant L. We suppose 
that the sequence fj, N weakly converges to a measure \x. Let t > 0. We want 
to prove that 



lim / f(x)^(dx) - / f(^(x))^(dx) = 

N— >oo 



The QSD property gives us that, for all k 



W(X n )]= / f{x)^{dx) 



E„ 



T N >k 



H N (dx) 



Let 



Then, for all k, 



f(x)tA N (dx) ~ J f^ t {x))^ N {d, 



f(x)v N (dx)- I f( Vt (x))^(dx 



fix?) 



T N >k 



H N {dx)- I f{<pt{x))n«(dx) 



f{X?) - f(<pt(x)) 



T N >k 



H N {dx) 



In particular, for k = [Nt]. 



E, 



f(X^)-fMx)) 



T N > [Nt] 



fi N (dx) 



By Theorem 3.1, we know that, for N large enough, we have 



\[D N (t) >5]^ 2de 



-8 2 cN 



Thus 

E x [D N (t)] 



+oo 



\[D N (t) >5]d5^ 



2de~ s2cN d5 = 
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From Theorem 3.3, we can infer 1 — pjy O [jjf 
This implies e dN ^ Nt ^ — 1 — > 0, which gives us the boundedness of 



e e N [Nt]_ 
Hence 



E x 



f(X^ m] )-f(vt(x)) 



¥ x [T* > [Nt]] 



> [Nt 
-H N {dx 



H N {dx) 



E, 



r I vN 
■ L l A [JVt] 



X t + X t - </? t (x) 



[2* > [Nt]] 



I 



E x [L(D N (t) + (Nt - [Nt])±(\\F\\ + T))] , 



[r^ > [jvt]] 



+r 



r [ATt] 



4. Support of the limiting measure. 

Let L = L({pn}) denote the limit set of the sequence (pn)n£N for the 
weak* topology. In view of Theorem 3.4, L consists of invariant measures. 
As the QSD have their support inside A, it is natural to study whether the 
limiting measure also take their support in A. However, by the Poincare 
Recurrence Theorem, every fi £ L is supported by the Birkhoff center 



BC(tp) = {x £ A ; x G oj(x)} 

Since the boundary of A intersects the Birkhoff center (e.g the vertices of 
the simplex are equilibria and thus inside the Birkhoff center) , knowing that 
the QSD converges to an invariant measure is not enough, we have to further 
study the support of the measure \x to ensure that it is strictly inside the 
interior of the simplex. For that we will need large deviation assumptions. 

Hypothesis 4.1 : 

For all a > 0, there exists a function 



S : V n X 



xC x ([0,T],V a 
{x,T,4> 



S(x,T, 
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with the following properties, where V a = A \ N a (dA), C([0, T], V a ) is the 
set of continuous functions ip from [0, T] to V a and C x ([0, T], V a ) is the set of 
continuous functions tj; from [0, T] to V a such that ip(0) = x, both equipped 
with the topology of uniform convergence. 

• For every s g]0, oo[ and T > 0, the set 

{<j>eC x ([0,T\,V a ) s.t. S(x,T,<fr) <a} 

is a compact set 

• For x e A and T > 0, S{x, T,^)=0^^ = F(<f> 8 ) Vs G [0, T] 

• X satisfies a large deviation principle with rate function S and speed 
1/N uniformly in x on compact subsets ofV a , 

i.e. for K compact subset ofV a , T > and A C C x ([0,T],V a ) 

- sup inf S(x,T,4>)^ liminf — log inf P X [X N G A] 

x&K(j>&A N-^oo N x€K 

limsup — log sap¥ x [X N G A] < - inf inf S(x,T,<j>) 

N-t-oo N xeK ' X ^ K <t>£A 

In particular when K = {x} we get the "classical" large deviation 
principle 

- inf S(x,T,<f>) < liminf —log¥ x [X N G A] 
limsup— log¥ x [X N G A] < - inf 5(x,T,0) 

• S is linear with regards to the concatenation of functions, i.e. ifT<T 

S{x,T,4>) = S(x,f,(f)\ [0<f] ) + S{<j)(f),T - f,4>\ [f T _ f] ) 

• lim S(x,T, (j>) = uniformly in x £ V a and (p G C x ([0,T], Vq,). 

Hypothesis 4.2 : 

Vc > 3U C an open neighborhood of dA such that 

lim inf — log¥ x \X N (l) G dA] > -c 

Example 5: 

Before going further we will verify that Hypotheses 4.1 and 4.2 hold for the 
nearest neighbor random walk model introduced in Example 1. 
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To show that Hypothesis 4.1 holds we will use Theorem 6.3.3 in [9] 
with Conditions 6.2.1 and 6.3.1, which will gives us a Laplace principle for 
-X" jv |[o,t] that holds uniformly on compacts, and Theorem 1.2.3 in [9] will, 
in turn, give us the desired uniform on compacts large deviation principle. 
We only need to show that Conditions 6.2.1 and 6.3.1 holds 
When transcribing our model in [9] setting, we get, for x G V a 

H(dy\x) = ^2pij(x)5 ej -e t (dy) + I 1 - ^Pij(x) J S (dy) 

We define 

H fl (x,a)=log exp(a,y)/i(dy\x) 

and 

L fJi (x,P)= sup {{a, (3) - H^(x,a)} 

aeR d 

We refer to Chapter 6.2 in [9] for elementary properties of these functions. 
Definition 4.3: 

n(dy\x) is said to verify Condition A (called 6.2.1 in [9]) if 

(i) For each a G IR d , sup H^(x,a) < oo 

x€R d 

(ii) The function mapping x G M. d h > is continuous in the topology 
of weak convergence. 

Definition 4.4: 

/j,(dy\x) is said to verify Condition B ( called 6.3.1 in [9]) if 

(i) The relative interior of the convex hull of the support of fj,(dy\x), 
Ri(Conv(Supp(n(.\x)))) , doesn't depend on x. 

(ii) G Ri(Conv(Supp(fi(.\x)))) 

Let a > 0, we'll now prove the L.D.P. on V a The conditions, as they are 
written, demand that x may take values in all of W 1 while, in our model, we 
only use A. To remedy to that we'll first embed our d-dimensional simplex 
in M. d ~ l and then extend fi to a kernel rj defined on all of M.^ 1 by taking 
r)(dy\x) = fj,(dy\p a (x)) , where p a is the convex projection on V a . 

This way we have a probability kernel rj that is defined on all of it 
is then easy to verify that Conditions A and B hold for rj. We thus get a 
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LDP for rj on all of R d_1 with speed 1/N and rate function 

fT 



S?7 



f L v (4>(t),(j)(t))dt if (ft is uniformly continuous 
otherwise 



We now remark that, when x E V a ,r](dy\x) = fi(dy\x) and thus H v (x, v) = 
H^(x,v) and L v (x,u) = L^(x,u). 

From that we deduce that, if (ft G C([0,T],V a ), then L v ((ft,(f)) = L^cj), (ft) 
and finally S v (x,T,(ft) = S^(x,T(ft). 

We finally get that Hypothesis 4.1 holds for our nearest neighbor random 
walk with the following rate function. 



Jq L^cftit), <p(t))dt if (ft is uniformly continuous 
otherwise 



S(x,T,<ft) 



We still have to verify that the wanted properties holds for this rate function: 

• From Proposition 6.2.4 in [9] we get that, for every s g]0, oo[ and 
T > 0, the set 

{<fteC x ([0,T],A) s.t. S(x,T,<ft) ^s} 

is a compact set. 

• Let x S V a , T > 0, it is already known that 

S(x,T,<f>)=0 & 4> s = F{cft s ) VsG[0,T] 

• The LDP comes from Theorem 6.3.3 in [9] 

• The linearity of S(x,T,(ft) follows easily from it's definition as an in- 
tegral. 

Let's now prove that Hypothesis 4.2 holds too. 
Proposition 4.5 : 

Suppose that, for every couple € {1, • • • d} 2 there exists k £ N such that 
d^Pi ■ (•£) 

D -rr — 7^ on {x 6 A ; X{ = 0}. Then Hypothesis ^.2 holds 

dx i 

Proof : 



We want to ensure that, for all c > there exists an open neighborhood 
U c of <9A such that 

lim inf —log¥ x \X N (l) £ dA] ^ -c. 
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We know that qt{x) and Pi(x) go to as Xi goes to and that there exists 

k G N such that — — % —r — > on {x G A ; X{ = 0} and thus — —^-7 — > a on 

oxf ' oxf 

a sufficient small neighborhood of {x G A ; = 0} with a > 0. 
Let 1 > 6 > and let x G U c = {x ; 3i Xj < 6}. We have 

P*[i-"(1) GdA] >V x [X$ b] G<9A] 

[Nb] 

> n ^ x(j) ) 

3=1 

where {x^)j is a sequence of points in A at such that xf = 4r. 
If b is small enough we have, for j ^ [Nb] 



Then 



[JVb] Dvri.li 



Thus 



1- - ^m.»A,^fM, 



inf -Zo^LY^l) G SA] ^(log(a) - 1) 



[Nb] , [Nb] 1 , , 

where C is a constant 

As iV goes to infinity, the right-hand term goes to Cb(log(a) — l)+Cblog(b) 
which is greater than — c for b small enough. Thus, for N large enough 

lim inf — log¥ x [X N (1) G dA] ^ -c 

Hence our random walk model satisfies Assumption 4.2. 



Finally both hypotheses holds for our model. 



Definition 4.6: 

We define 



C(x, y) = lim sup inf S(x,T,t 

T^oo 0GC4[O,T]),</,(T)=y 
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We will say that x £-leads to y (denoted by x ~^£ y) if, for every e > 0, 
there exists a path of points x = £o> £1, ' ' ' £n(£) = V such that 

n(0 
fc=l 

We define B c (x,y) = inf 

5 linking a; to y 

Thus x ~»£ y iff Bc(x, y) = 

We will say that x is £-chain recurrent if x ~*>£ x and will denote by 
TZ-civ) the set of all £-chain recurrent points. 

If x and y are two points of TZc verifying x ~*>£ y and y -^>£ x we will 
then denote x ~£ y. The equivalence classes for this relation will be called 
£-basic classes. We define a partial order on these classes by [x] -<£ [y] if 
x -w£ y. A maximal £-basic class will be called a £-quasi-attractor. 

Hypothesis 4.7 : 

There is only a finite number of C-basic classes in A denoted by Ki , i = 
1 • • • v. We suppose that they are closed sets and indexed in such a way that 
the k first {K{\i = i...k are the C- quasi- attractor s and the v — k others aren't. 

Proposition 4.8 : 

The function C has the following properties : for every sequence (x ra ) ng N £ 
V a converging to an x G V a and every y G V a we have 

lim C(x n ,y) = C(x,y) lim C(y,x n ) = C(y,x) 

n— >oo n— >oo 

Proof : 

This proposition follows easily from Hypothesis 4.1 



The following theorem is the main result of this section, giving us more 
insight in the support of the limiting measure /i. 

Theorem 4.9 : 

We suppose that the flow {(ft} associated with the mean dynamic x = F(x) 
has an interior attractor. Under Hypotheses 4-1, 4-% an d 4'% we have : 

The limiting measure \i has its support in the union of the C-quasi- 

k 

attractor, i.e. in Ki 
i=i 
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We will prove this theorem under an intermediary set of hypotheses then 
we will prove that the announced hypotheses imply the intermediary hy- 
potheses. 

4.1. Absorption-preserving pseudo-orbit. 
Here we introduce a different notion of chains for our dynamical system 
using absorption preserving 5, T pseudo-orbit, an analog to 5, T pseudo- 
orbits introduced by Conley [8], which have been extensively studied in the 
past. 

Definition 4.10: 

Let {ipt}t<=R be a flow given by an ordinary differential equation on (A,d) 
for which dA is an invariant set. We will call (5, T) absorption preserving 
pseudo-orbit (8, T-ap-pseudo orbit) from x to y a piecewise continuous path 

x = XQ,{(p t {xi) ; t g [0, ti]}, {<ft(x2) ; t g [0, t 2 ]} 

■■■{ift(x 1 );te[0,t k ]},x k+ i = y fe>l 

which is uniquely defined by the sequences of points Xo, ■ ■ • and times 
t\, ■ ■ ■ tk such that the following hypotheses hold: 

d(x, x\) < 5 

d((p tj (xj),x j+ i) <5 Vj = 1 • • • k 

ti ^ T \/i = 1 • • ■ k 

xj G dA =>■ xj+i G dA Vj = • • • k 

We will denote then x ~-^> a p,6,T V 

If x ~^>ap,s,T V for every 5 > and every T > 0, we will denote x ~^> ap y 
The point x will be said to be ap-chain recurrent if x ~^> ap x, we define 
lZ ap ((p) the set of ap-chain recurrent points. 

If x and y are two points of TZ ap (ip) such that x ~^ ap y and y ~^ ap x we 
will write x ~ ap y. The equivalence classes for this relation will be called 
ap-basic classes. We define a partial order on these classes by [x] < ap [y] if 
x ~^* a p V- A maximal ap-basic class will be called a ap-quasi-attractor. 

The ap-chain recurrent points and the ap-basic classes have some inter- 
esting properties which will be of use when proving the result on the support 
of the limit measure /i. We enumerate and prove some of them. 

Proposition 4.11 : 

Let x G TZ ap . Then [x] ap C [x] ap U dA. Moreover, for all t > [x] ap is ipt 
invariant. 
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Proof : 

Let y € [x] and let (yk)keN be a sequence of elements of [ x\ a p converging 
to y. Suppose y dA. 

Let 5 > and T > and let k such that d(yk,y) < 5. 

There exists a 5, T pseudo-orbit x, (xi,t±), • • • , (x n , t n ),yk linking x to y^. 
Thus x, (xi,ti), • • • , (x n ,t n ),y is a 25, T pseudo-orbit linking x a y. 

Hence x ~^ ap y. 

A similar reasoning on 5, T-pseudo-orbit linking y^ to x gives us y -^ ap x 
and then y S [x] ap . 
We show now that f 

x\ap is an invariant set. 

Let T,T',e > 0. fx' is an uniformly continuous application. Let then 
5 < e such that d(x,y) < S implies d(ipT' (x) , <pt' (y)) < £■ 

We know that there exists a 5, T pseudo-orbit x, (xi,t\), ■ ■ ■ ,(x n ,t n ),x 
linking x to x. Hence x, (x±, t±), ■ ■ ■ , (x n , t n + T'), ipT'(x) is a e, T pseudo- 
orbit linking x to <Pt'( x ). 

Before proving the converse, let's just remark that, if 5i < 62 and T\ > T2 
then every 8\,T\ pseudo-orbit is also a 82^2 pseudo-orbit. 

Let's now suppose that T > T' 

We consider again our 5, T pseudo-orbit x, (xi,t\), • • • , (x n ,t n ),x linking 
x to x. One can remark that t\ + ti — T' > T. 

Thus (pT'(x), ((pT'(xi),t 1 +t 2 -T), (x 3 ,t 3 ), • • • , (x n ,t n ),x is a e, T pseudo- 
orbit linking Lpx' (x) to x if 5 is chosen small enough. 

Then x ~^ ap ipT>{x). 

By composing the 5, T pseudo-orbits linking x and ip?' (x) by (f-T' we get 
that x ~ ap (pr(x) for every T in R. 



Proposition 4.12 : 

Let x £ A. If x £ dA or oj{x) C A then uj{x) C TZ a p- From this we get that, 
for every x in A, uj{x) n lZ ap 7^ 0. 

Proof : 

The first point is a well-known result for "classic" chain-recurrence and 
easily extended to ap-chain-recurrence. Furthermore, if x £ A and lo(x) n 
dA 7^ then, by taking y € w(x) n<9A, we get w(y) C oj(x) and w(y) C 7£ ap . 



Proposition 4.13 : 

If [x]ap is maximal, then x ~-> ap z if and only if z G [x]. ^4s a consequence 
we also get that every quasi- attractor is a closed set. 
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Proof : 

This result is trivial as soon as z G 7Z ap , thus we only have to prove it for 
z TZap- We know that ui(z) n 1Z ap ^ and that, if u G uj(z) n 7£ ap then 
-2 ~^a P w thus x ~^ ap u. Hence u G [x] ap , from that We get X ' w ap Z ~^ap 
u ^ap s which implies z G [x] ap C 1Z ap . 

■ 

The relation between being an attractor and being a quasi-attractor has 
been studied in the past, we recall this theorem of [1]. 

Proposition 4.14 : 

Let C be a non-empty subset of A. The following assertions are equivalent : 

(i) C is an irreducible attractor i.e. it doesn't contain any proper attractor. 
(ii) C is an isolated quasi-attractor, i.e. there exists U, an open neighbor- 
hood of C, such that U n 1Z ap = C. 
(Hi) C is an isolated connected component oflZ ap and 

C + = {x£ M;3y£TZ ap \C s.t. C ^ ap y ^ ap x} = 

This result is proved in Part 5 of [1]. 

The following hypothesis is an analog of Hypothesis 4.1 adapted to the 
context of ap-pseudo-orbits. 

Hypothesis 4.15 : 

There is only a finite number of ap-basic classes in A denoted by Ki , i = 
1 • • • rj. We suppose that they are closed sets and indexed in such a way that 
the k first {-Ki}i=i...fc are the quasi- attractor s and the rj — k others aren't. 

Proposition 4.16 : 

For every 9 small enough, there exist 5(9) G [0, 9[ and T(9) G]0,oo] with 
6(9), T(9) ^ (0,oo) such that: 

If there exists a 8,T pseudo-orbit £o • • • £n verifying, 5 < 5(9) orT > T(9) 
and, for a certain triplet (i, i',j) G {1 • • • n} 3 , 

d(£o, K) < 5 d^fy) < 5 d(Cj,Ki) > 9 

Then i ^ i' and Ki -< Ky . 

This proposition means that, for 5 small and T large, the 5, T pseudo- 
orbits respect the partial order. 
Proof : 
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Suppose that, for every 5 > and every T > 0, there exists a 5, T-pseudo- 
orbit Co ■ ■ ■ Cn such that 

d(£o,tfi)<<$ d(£„,#i')<* 

then, we can construct 5, T-pseudo-orbits going from Ki to Ky which in turn 
implies K { <^ ap K v 

Hence, if Ki -/+ ap Ky, there exists 5^0 and T g]0, +oo] such that a 5, T 
pseudo-orbit Co • • • Cn verifying 

d^ ,Ki)<S d(Z n ,Ki,)<6 

may only exist if 5 > 5 or T < T. 

Suppose now that i = i' , we will show that there exists S and T such that 
every 5, T pseudo-orbit Co ■ ■ ■ Cn verifying either 5 < 5 or T > T and 

d{^,Ki)<6 d(C n ,Ki)<5 

doesn't contain any point at a distance greater than 9 from Ki. 

Suppose first that this assertion is false, thus we have real sequences 5i — > 
and Ti — > oo and a sequence of <5/ , 7} pseudo-orbits verifying 

d(&,Ki)<6 d(e ni ,Ki)<6 d(£ l jl ,K i )>0. 

A being a compact set, we may suppose that, up to an extraction, as I 
goes to infinity x l — > x G Ki, x l ni — > z G Ki and — >■ y with Kj) ^ 9. 
In that case we get x ~^> ap y ~^> ap z and thus y ^ Ki, which is absurd. 



Proposition 4.17 : 

For every 5 > 0, there exists Tq > such that every 5, Tq pseudo-orbit 
intersects N s (7Z ap ). 

Proof : 

Let x G A and 7 > 0, we define T>(x) = Inf{t ^ ; ip t (x) G N~>(K ap )} 
As uj{x) n TZ ap ^ we get T~<(x) < +00. 

For a > we will denote iV a = {x G A ; T 7 (x) ^ a} the level sets of T 7 
Let us show that 7V a is closed. Let (x n ) ng N be a sequence of elements of 
N a converging to y. By the continuity of (ft we obtain then that, for every 
t > 0, lim ip t (x n ) = ip t (y)- If t < a we get (p t (x n ) G (iV 7 (^ ap )) c which is 

closed. Hence we have <£>t(y) ^V 7 (72. op ) for all t < a and thus y G iV a . 

T"'(x) is then an upper semi-continuous function taking its values in 
[0, +00 [, A being a compact set, we know then that T 7 (x) attains its maxi- 
mum on A which we will denote T 7 . 

Taking Tq > T s gives us the result. 
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The following corollary comes easily from the last two propositions. 
Corollary 4.18 : 

For every 5 > and every T > there exists a family Vi of open neighbor- 
hoods of the Ki and positive real numbers 5\ and T\ such that 



• N Sl (Ki) c Vi fori = l---n 

• Every 5\,T\ pseudo-orbit starting from Vi stays in Vi for i = 1 ■ ■ • k 

• If there exists a 8\,T\ pseudo-orbit x, (£i, t±), ■ ■ ■ ,(£ p ,t p ),y with x G 
N Sl (Ki) and y G N Sl (Kj) such that 

31 3t G [0, t t ] such that ipf(€i) V 

Then i / i! and Ki -< K# . 

• Every 5\,T\ pseudo-orbit intersects N s (JZ ap ) 

Definition 4.19: 

For K compact subset of A we denote 

fc, K (N)= sup F X [X N (1) G A\N\ Vl (x))] 
xeA N nK 

Proposition 4.20 : 

If the flow {ipt} admits an attractor Ac A, then, for all K compact subset 
of A and neighborhood of A, there exists 5 > such that p 1 ^ ^ 1 — Ps,k{N). 
Moreover, if there exists Vk an open neighborhood of dA such that 

lim = 

^oo inf^nAjv Vx[X N (l) G OA] 

Then p(V K ) = 0. 
Proof : 

Let U be an open neighborhood of A and 5 > such that U C K and for 
every t, ip t (U) C U and N s ((pi(U)) C U. 



QSD FOR STOCHASTIC APPROXIMATION ALGORITHMS 23 

Then 



x&A N 

x et^A N xeUnAN 



^H N (U)[l- sup p%{x,U c ) 



>H N (U)\l- sup ^(x,(AT 5 (^(C/))) c ) 

.AT 



^^(C/)(1-^(AT)) 

We finally get p»>l-p s {N) 
From this we obtain 

l-^(iV)^^(A) 

< E (l-^(^9A))^(x) 

xeA* N 



Hence 



^(M\1^)+^(Vk) 1- mf p$(x,SA) 



inf^eVonAjv f^O'^A) 

Vq being an open set, the weak convergence of the measures ^ gives us 
the desired result. 



Hypothesis 4.21 : 

We suppose that the flow ipt admits an attractor A C A and that there exists 
K a compact neighborhood of A in A and Vk an open neighborhood of dA 
such that 

Hm = 



illf I6 ^.nA„ P,[X"(1) € dA] 



The following assumption is a technical one but we will see later that it 
is true under the first set of hypotheses. 
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Hypothesis 4.22 : 

Let j G {k + 1, • • • , v}, i.e. such that Kj is not a quasi- attractor. Then 
3r]>0 V7 > 3N 3x ■ N -> R 3( : N -»• R 

lim C( n ) = 1™ x( n ) = 

and 



sup F x 

xGJV7(Xj) 



We now arrive at the central theorem of this section. 
Theorem 4.23 : 

Under Hypotheses 4-15, 4-21 and 4- 22, the limiting measure fx has its support 

k 

inside the union of ap- quasi- attractor s K{ 

i=i 

Proof : Let 5 > small enough (how small will be specified later) 

K being an attractor, we know that pn — > 1, that /i is (^-invariant and 

has its support inside 1Z ap . 

Hypothesis 4.21 and Proposition 4.20 gives us that /x(<9A) = 0, hence 

H(K) = 1. 

It only remains to be shown that, for every j = k + 1 • • • 77, there exists an 
open neighborhood Wj of Kj such that piWj) = 
We know that 

PN > (t N (Wj) = V F X [X(N) G Wj] = [ F X [X% G Wj]dfi N (x) 

177 r JM 



We denote f N 



xeM 

N 

i 



and define the following events : 

E S N = |3<5i,Ti pseudo-orbit closer than 5 from , t G [0,iV]| 

E' N = {Vi G {k + 1, •••,??}, Vg > iV 2 G ^ (i^) => ^1 J 

-E^y is the event "after its first entry in N Sl (Ki) the Markov chain X will 
have left it after iV 2 steps". 

Let Wi be open neighborhoods of the Ki such that N s (Wi) C Vi for every 
i = 1 • ■ • rj. On E s N t~)E' N we get, 
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For j ^ k, 



Hence 



X% G Wj =► Vi G {2, • • • b} X$ N d (K) 



N 

i 

N 



F[x% g Wj\E%nE' N } ^ 



i=2 



On E & N , starting from N Sl S (K{) the chain cannot enter N Sl S (K{) once 
it has left Wf (and also Vi). 
Thus 

F[(E' N ) C \E S N ] < P[3i such that X doesn't leave Wi before the time N 2 ] 

v 

< V sup P,[f^ > iV 2 ] 

< V sup P x [i^iV 2 ] 

l=fc+ lrtw*l-*(^) 

where = Inf{t > ; X? &U}. 

We now remark that, if A, -B and C are three events, we get 

P[C] = P[C nfi c ni] + P[CnBnA] + P[C n 
< P[s c n A] + P[C|s n vl] + P[^l c ] 

^ F[B C \A] + F[C\B n A] + F[A C ] 
From that, we obtain 

V*[X% G W,-] < Px[(^at) c ] + P x [(^) c |^] + W X [X% G n 

It only remains to control P X [(S^) C ]. In order to do that we will consider 
the pseudo-orbit PO N (t) defined by x, (x,Ti), (X^,Ti) ■ ■ ■ 

For N large enough, classic results on stochastic approximation algorithms 
(see e.g. [3]) give us the following estimate : 

F X [(E S N ) C ] sC F[PO N (t) isn't a Si,T x pseudo-orbit] 

+ ¥ x [ sup d(X t N ,PO N (t))>5] 

te[o,N] 



<0((1 



-eN\N 



)») = 0(e Ne ~'") 



2G 
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with e > 

In conclusion we get 



Pat ja 

< -L /* p.k^i + p a: [(^) c |^]d/i JV (x) 

+ ^ fF x [Jt^€ Wj\E s N n i^d^O*) 

< 4 fo(e^- £iV ) + C(^) + E^> iV ((^W) C )) 

*S 4 (0(^ e_SjV ) + C(iV) + bv N ((N s (K)y)) 
Pn V 7 

From Theorem 3.3, we can infer that 4 — >■ 1. 

Hence we have limjv->oo C^j) = 0. As the sets Vj are open neighbor- 
hoods of the sets Kj we obtain fi(Kj) = 0. 



4.2. Going back the hypotheses 4-1, 4-% an d 4-7- 
The aim of this section is to prove that our first set of hypotheses implies 
the second one, thus proving the announced Theorem 4.9. In particular we 
will show that £-quasi-attractors and ap-quasi attractors are the same. In 
this section we will assume Hypotheses 4.1, 4.2 and 4.7 to be true. 



Proposition 4.24 : 

For every compact set K C V a , there exists a open neighborhood Vk of <9A 
such that 

lim ^1 = 

N->oo inW^A* G dA] 

where Ps,k( N ) = SU P ^x[X N (l) 2 A^lO))] 

x&KnA N 

Proof : Let K be a compact set in and let 5q > such that 

5n < inf d(ipt(x),9A) 

t(£[0,l],x£K 
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Let c(K) = llnf{S(x,l,<p)\x G K,d(4>,<p(x)) ^ 6 } > 0. 
For x G K we have 

1 



lim sup —logF x 

JV-S-oo iv 



X" e {<f>eC x ([0,l]);d(cf>,<p(x))>6o} 



<- inf S(x,l,<f>) 

4>;d(4>,(p(x))>5o 

< -4c(K) 

Thus, for iV large enough and for every x in if . 

l/o^P* [F € {</> € C x ([O,l]);d(0,^(x)) > <5 }] < -3c(if) 
Moreover, there exists Vr-, an open neighborhood of <9A such that 

lim inf — log¥ x [X N (l) G <9A] > -2c(K) 
Then, for iV large enough, we have, for every x in Vr 



Finally we get 



lim 



F X [X N {1) G 9A] > 



N^oo mf xeVK F X [X N (1) G a A] N^oo 



s£ lim e V N = 



Proposition 4.25 : 

Lei T > and K be a compact subset of V a . Then, for every 6 > 0, there 
exists e(K, T, 6) such that 

\/xeK S(x,T,<p) < e d(<f>,(pXx))[ 0iT \ < 8 

Proof : Let's suppose that this result is false, then 

36 Ve 3x 3<f) £ s.t. S(x,T,(f> e ) < e and d(ip. (x), 4> £ ) > 6 

We also know that c = m({S(x, T,<j>) ;x G K ; d(<p, (p. (x)) > 6} > 
Thus, for every e > 

e > S(X,T,<f> e ) > c> 



which is absurd. 
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Proposition 4.26 : 

The function Be '■ V a x V a — > R+ is upper semi- continuous. 

Proof : Let x n and y n two sequences in V a converging to x and y in V a 
For every 5 > there exists a path x = £o, ■ ■ ■ £ n /£\ = y such that 

B c (x,y) ^A n{0 (0 ^B c (x,y) + 5 

Let's now consider the path £ n given by x n = €q,£i,--- ^)-i,y n = £n(£) 
If we take n such that 

|£(^n,6) - £{x,€i)\ < £ and \£(€ n (s)-i,y n ) - ^(Cn(O-i'y)l < 6 
We obtain 

Bc{x n ,y n ) < ^n(0(^) < ^(s.y) +35 
Thus, for every 5 > 0, limsup/3/;(x n ,y n ) ^ Bc(x,y) + 3<5 

n— >oo 

Finally limsup 73/;(x n , y„) ^ B c (x,y) 

n— ¥oo 

m 

Proposition 4.27 : 

Let x G TZc such that [x]c is a closed subset of V a . Let 8 > such that 
N e ([x]c) C V a . 
Then, 

35 > 3T > with *(0) G [x]/;, *(f ) G [x] £ , f > T, 

andS(V(0),f,V)<6 Vt€[0,f] tf(t) G iV e ([x]/;). 

Proof : Let x G 72./; such that [x]/; is a closed subset of V a and let 6* > 
such that N e ([x]c) CV a . 

Suppose that the announced result is false, then, there exists a family of 
functions \I> n G C([0,T n ],V a ) such that 

*n(0) G [x] c *n(T n ) G [x] c lim T n = oo 5(*(0), T n , *) < 1/n 

n— s-oo 

and 

Vn 3t„ > such that * n (i n ) G" iV e ([x]/;). 

[x]/; being a compact set, we can assume without loss of generality that 
*„(0) -»• u G [x] £ and tf n (T n ) 4»£ [x] £ . 
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Let T n = in£{t > ; * n (i) G" N e ([x] c )}- Then we have 

^n(r„) eVa\N e ([x] c ) 

and p^j \ TV ([x]^) is a compact set. 

Thus, without loss of generality, we can assume that 

*n(r„) -^wGV a \ N 9 ([x]c). 

Using the sequential continuity of C and the fact that u and v are C- 
chain-recurrent points, we get that u ~^£ w and w ~^£ v. Thus w G [x]£ 
and we get a contradiction 



Corollary 4.28 : 

Suppose TZc is a closed subset of V a . Let 9 > such that N e {lZc) C 
T/ien ; 

35 > 3T > V* ifltifc (0) G *(T) G He, f > T 

and S(¥(0),T,¥) < 5 ViG[0,f] ^(t) G N 6 {1Zc). 

Proposition 4.29 : 

Let x G 7?.£ smc/i i/iai [x]£ is a closed subset of V a . Then x G 7£ ap and 
[x]£ C [x] ap . 

Proof : Let y G Let (T n ) 

neN be a sequence of positive real numbers 
and ^ n be a sequence of functions from [0,T n ] to V a such that ^ n (0) = x, 
tf n (T n ) = y and S(x,T n ,^ n ) < 1/n. 

T n is either a bounded sequence or it goes to infinity (up to a sub- 
sequence) . 

Let's suppose that (T n ) n€ ^ is an increasing sequence of positive real num- 
bers going to infinity cind ^ n a sequence of functions such that ^(0) = x, 
^ n {T n ) = y and S(x,T n ,^ n ) < 1/n 

The former proposition gives us a compact set K C V a , and positive 
numbers 8 and T such that, for T n > T and 1/n < 5, \l/ n lives in K 

Let T > T, 5 < 5 and e = e(5, T, K) given by Proposition 4.25. 

Then, from Proposition 4.25, we can infer that, for n large enough, 

As ^„(T) G K, we also have 



d(^(*„(r)),* n (2f))<<5 
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By iterating this process we get that, for n large enough, 

(x,T),(* n (T),T),--- 

is a 8, f ap-pseudo orbit linking x to y. 
Hence x ~^ ap y. 

Let's suppose now that T n is bounded by T. By taking a sub-sequence 
we can assume that T n — > T as n — > oo for some T with T ^ T ^ 0. 
we continue the function ^ n to the time T by concatenating the flow <p m . 
Then we know that ^ n — > (f.( x ) uniformly over [0, T] as n — > oo. Hence 
y = lim^oo *„(T n ) = ipr(x). Thus T > and y € 7 + (^)- 

If y = x we get then that, either 

% or x is periodic which implies 

X G TZ a p. 

Then, if y G [x]c we obtain that either x ~^ ap y or y G 7 + (x) C [x] ap - 
Taking y = x gives us x G 7£ ap , i-e. TZc C 7£ ap . As the roles of x and y can 
be exchanged we also get y ~^ap % and in conclusion \x\c C [x] a p. 



Proposition 4.30 : 

Closed C-classes in V a are positively invariant sets for the flow ip 

Proof : Let x G TZc such that [x]c is a closed C basic class in V a . Let T > 
and 9 such that N e ([x] c ) CV a . 

If the paths linking x to itself have bounded length then x is periodic and 
in this case 7 + (x) C [x]c- 

We now suppose that the paths have unbounded length. 

Let e = e(\x]c,T,9) and let 5 > and r > given by Proposition 4.27, 
let $ be a path of length greater than r and of cost smaller than min(£, S) 
linking x to itself. 

Then, by the triangular inequality we get 

d(ip T (x), [x]c) < d(ip T (x), *(T)) + d(*(T), [x]c) ^ 29 
Thus, by making 9 go to zero, we obtain (pr(x) G [x]c = [x]c] 



Lemma 4.31 : 

Let ^ be a binary, transitive relation on V a , such that, for every x G V c 
every y G a(x) and every z G oj{x), 

y ^ x x >— > z z >— > z 
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Let [x]^_> be a maximal, closed and isolated ^-basic class. 
Then [x]^ is an attractor for the flow if 

Proof : Let W be an isolating open neighborhood of [x]^, i.e. W is an open 
neighborhood of [x]^ such that the only ^-recurrent points of W belong 

to [#],_>. 

If [x]^ isn't an attractor then there exists p G dW such that 

7-(p) C W and a(p) C 

Let y G co(p), we have p >— > y and y y. 

Similarly let z 6 a(j)), we have z >— ► p and thus z >— ► y. 

Let's define 

[x]^ = {z , Ely G 7£ M \ and x >— » y >— > ^} 

As [x]>_, is maximal, we have [x]+ =0. 
Yet we have x >— ► y and y G 

Hence y G [&]>->, and, as x >— > p s— > y we get p G [x]^_, which is absurd. 



Proposition 4.32 : 

Let [x]c be a closed isolated quasi- attractor. Then [x]c is an attractor. 

Proposition 4.33 : 

Let K be a compact subset ofV a . 

For n > 0, 5 > 0, T > there exists Nq > such that, for all N > Nq, 
allxe K, allT <f and all * G C x ([0,T],K) 

F x [d(V,X N )< V }>exp + * 



1/N 

Proof : Let tf T = sup{\S(x,T,^) - S(x,T,</>)\ ; x,y G K , d(^,<p) < 7}. 
The inferior semi-continuity of S gives us that lim & T = 

Let 7] > 0, 5 > 0, f > and let 7 < n such that < <5 and N J (K) C F a 
The large deviation principle gives us that, for 4> & C x ([0,T],K), 

liminf hog wf¥ x [X G 7VJ(</>)]] ^ - sup inf S(x,T,V) 



where 7V 7 (^>) = € C([0,T],7Y) ; < 7}. Thus, there exists y^(iV) 

a function which goes to zero as N goes to infinity such that 
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hog M ¥ X [X G JVJ (<£)]] ^ - sup inf S(x,T9) - g K {N) 
Hence 

P x [d(^,X N ) < V ]> F x [d(*,£ N ) < 7] ^ inf P X [X G JVW] 



^ exp 



1/N 

-Z« T -S{x,T,*)-g K {N) 



1/N I 

Taking N large enough such that $ T + gx {N) < 5 gives us the result 

I 

Proposition 4.34 : 



We denote by BC((p) = {x G A ; x G ui(x)} the Birkhoff center of the flow 
ip. Then 

V a n BC(ip) C K c 



Proof : We know that BC{ip) = Rec{ip) = {x G A ; x G uj(x)}. It's appar- 
ent that Rec(ip) C 7?.£. Hypothesis 4.7 allow us to conclude. 



Corollary 4.35 : 

Let 11 be an invariant measure for the flow (p whose support S lies within 
V a . Then S C IZc- 

We arrive at the main theorem of this section, linking £-chain recurrence 
with ap-chain recurrence. 

Theorem 4.36 : 

Under the hypotheses 4-1, 4-% an d 4-7 we have 

TZ ap C\V a = K c C\V a 

Vx G IZc H V a [x]c = [x]ap 

Proof : By Proposition 4.29 we already know that, if [x]c is a closed £-basic 
class in A, then x G 1Z ap and [x]c C [x] ap . 

The function Be is upper semi-continuous. Thus 

V7 > 3U1 neighborhood of K t s.t. V(a, b) G (Ui,) 2 B c {a, b) < 7 



QSD FOR STOCHASTIC APPROXIMATION ALGORITHMS 33 



Lemma 4.37 : 

Let U be a neighborhood of TZc H A such that K = U is a compact subset 
of A. Thus there exists T > such that, ifj + (x) C K then {(ft{x) ; t G 
[O,T]}nC/^0 

Proof : Due to the fact that, if j+(x) C K, u{x) C U we can't have 
7 + (x) C K\U. Let's define z/(x) = m/{t ; ipt(x) $ K\U} < +oo 

We will show that v(x) is upper semi-continuous. 

Let x £ K and x n G if N such that x n — )• x. 

The continuity of </9 gives us d(<pT n (x n ), cpt(x)) — > with the convention 
that <^oo(x) = u(x). Let T = i/(a;), there exists a sequence e m — >■ such that 
<p f+£ Jx)?K\U. 

Yet we have ff +e (x n ) — > <ff +E (x). Hence, for n large enough we get 
v{x n ) < f + e m 

We now get lim sup v{x n ) + e m 

n—>oo 

Then lim sup ^(x n ) ^ v{x), i.e. v is u.s.c. 

Thus T = max v{x) exists and we get, if j + (x) C K then 
xeK\u 

Wt(x);t G [O,T + e]}nC//0 

■ 

Let x G 7£ ap n A and let y G [x] ap , there exists two sequences of positive 
real numbers 5k — > 0, Tk — > oo and a sequence of 5k, Tk ap-pseudo-orbits 
linking x to y denoted *& k . 

Lemma 4.37 gives us K C A a compact neighborhood of 7£ ap such that, 
for k large enough, every 5k, Tk ap-pseudo-orbit stays in K. 

We know that K c C ^ ap C if. Let 7 such that |J N J (Ki) C if. Let then 

i 

T given by the former lemma. For k large enough, we have Tk > T and thus 
every continuous part of the 5k, Tk ap-pseudo-orbit intersect U 7 . 

Let s < 7 and V = N £ {Ki), we assume e to be small enough such that 

i 

the e-neighborhood of the K\ are disjoint. 

We define two sequences of times {ai{k))i = \... Pk and {Ti{k))i = \... qk by 

(7 (Jb) = T (k) = min{t ; tf£ y} 

ff<+l(fc) = min{t > Ti{k) ; ^ fe G V} r i+ i(fc) = min{t > a l+1 {k) ; ^ t fc V} 

By Lemma 4.37 we know that o~i+i(k) — Ti{k) < 2T. 
If our pseudo-orbit ip k enters more than once the same N £ (Ki) then we 
truncate what happens between the first entry and the last exit and we will 
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keep the same name for the new path (which may no more be a pseudo- 
orbit). This way we get q k ,p k < V- 

Hye\jKi then q k = p k + 1, else q k = p k . 

i 

In the first case we get 

9fc~ 1 Pk 
8=0 1=0 

In the second case, we get 

9fc~l Pk 

B c (x,y) < £ B4*i 4W ,*U) + E^«W.*i W (*)) + ^« fc »f) 

i=0 i=0 
In either case we obtain 

Bc(x, y) < (1 + z/)a + (1 + i/)5«p {-B £ (a, 6) ; d(b, V9[ ,2T] (a)) < £fe} 
where 

£k= sup d((ft{x),(p t (y)) 

d(x,y)<S k ,t£{0,2T] 

From the continuity of (p, the positive invariance of [x] ap and Proposition 
4.8 we infer that 

lim Sup{Bc(a,b) ; d(b, <f[o,2T] («)) < £fc} = 

Hence we have, for every a > 0, Bc(x, y) ^ (v + l)a. 
In conclusion x y. Similarly y ~->£ x, i.e. y 6 [x]^ 



Remark : This proof gives us a little more, it proves that, if y £ lZ ap and 
x ^ ap y, then x V- 

When we take a such that A n IZc = V a C\ TZc, this proposition proves 
that the first set of hypotheses implies Hypothesis 4.22. 

Proposition 4.38 : 

Let j G {k + 1, • • • ,r]}, i.e. such that Kj is not a ap- quasi- attractor. Then 
3A > V7 > 3N 3( : N -> R urai/i lim £(n) = sttc/i i/iai 



sup r x 

xeN x (K j ) 
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Proof : Kj isn't a quasi-attractor, thus 3 A > such that N 2X (Kj) C A 
and, for all 7 > and all x G N x (Kj), there exists T 7 and ^ such that 



* 7 (0)=x, f 7 (T 7 ) = y 7 N 2X (Kj), S(x,T 7 ,^ 7 ) < 7 



Let then J7 = N 2X (Kj), V = u(U). Let r > and let K be a compact 
subset of A such that N r (U) C K. 

As \I/ 7 starts in £/ and ends outside of U it must pass through K \ U. 
Without loss of generality we may suppose that ^ 7 stays in K and y 1 G 
K\U. 

Proposition 4.33 says that : 

ForA>0,5>0,T>0, there exists N Q (K) > such that, for all 
N > N , all T < f, all x G K and all * G C x ([0, T],K) 



F x [d(V,X N ) < A] > exp 



S(x,T,^) + 5 
l/N 



Applying this to our case gives us Nq such that, ViV > iV"o 



: [d(^,X N ) < A] ^ exp 



S(x,T^,^) + S 

TJn 



^ exp 



7 + A 



Yet {(1(1^, f 7 ) < A} implies that X N leaves N x (Kj) before the time T 7 
Hence 



VxGK VN>N Q {K) V x [t nHk . } >T 7 ] <l-e" 



Then 



_7+5 

< I 1 - e Viv 



27 

,T7lv 



TT 



2 7 
,TTN 



< e 



< e 



< e: 



TT 



27 

,TTN 



T1 



7 — s 
'TTN 1 
TT 



7+6 

in I l- e W 



7+<5 

TTN 



7 — 6 



Taking <5 < 7 and ((N) = e~ 



allows us to conclude. 
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We linked ap-basic classes with £-basic classes but, unless the partial 
orders on the class are similar, we might not have the same quasi-attractors 
for both partial orders. This proposition shows that the quasi-attractors are 
the same. 

Proposition 4.39 : 

Let [x] be a basic-class, [x] is a C- quasi- attractor if and only if [x] is an 
ap- quasi- attractor. 

Proof : 

Theorem 4.36 already gives us that, if y G 1Z ap and x ~^ ap y, then x y. 
Hence, if [x] is a £-quasi-attractor, it is also an ap-quasi-attractor. 
Suppose now that [x] is an ap-quasi-attractor. 

Let y G IZc such that x -^>c V- If the path linking x to y have bounded 
length we get y G 7 + (x). As £-basic classes are positive invariant sets we 
get then y G [x]. 

Let us now suppose that the paths linking x to y have unbounded length. 
Propositions 4.28 and 4.8 give us a compact set containing the paths linking 
x to y for T large enough. Using the same technique as in the proof of the 
proposition 4.29 gives us x ~^ ap y, i.e. y G [x\. 



APPENDIX A: REMINDERS 

The purpose of this appendix is to give small reminders about some basic 
properties of the objects used here. We won't give extensive proofs of the 
results or complete theory of these objects but will include some references 
should the reader want to know more. 

A.l. Quasi-stationary distributions. 

This subsection has been inspired by the notes of a course given by Sylvie 
Meleard at the VI Escuela de Probabilidad y Procesos Estocasticos in Gua- 
najuato in September 2009. 

Here we will consider a Markov chain Zt, either in discrete or continuous 
time, whose state space E cM. d admits an absorbing state, denoted by {0}. 
We will denote E* = E \ {0}, V* the set of probability measures whose 
support lies in E*. We define Tq to be the absorption time. 



T = inf{t > 0; Z t = 0} 



QSD FOR STOCHASTIC APPROXIMATION ALGORITHMS 37 

We suppose that, whatever the initial state is, the process will almost surely 
be absorbed, i.e. 

Vz G E F Z [T < oo] = 1 

Definition A.l: 



1. A probability measure \x on E* is said to be a quasi-stationary distri- 
bution (QSD) if and only if, for every Borelian set A C E* and every 
t > 0, 

F^Zt € A\T > t] = n{A) 

We remark that, in this case, \x is a fixed point for the conditional 
evolution 

vv+W v \Z t G .\T >t] 

2. A quasi-limiting measure (QLD) for a G V* is a probability measure 
f on E 1 * such that, is the weak convergence sense, 

lim F a [Z t G .\T Q >t] = v 

t— >oo 

3. The Yaglom limit is the probability measure ir on E* defined by, for 
A G B(E*), 

n(A) = lim ¥ z [Z t G ^|T > t] 

t— ¥00 

as soon as this limit exists and doesn't depend on z G E* 
Proposition A. 2 : 

Suppose that n is a QSD for this process Z%. Then there exists a positive real 
number 9(/j,) such that 

P^To > t] = e~ eMt 

See [15] for a proof 

The following theorem is true in a broader context (see e.g. Sylvie Meleard 
course notes or Bonsall paper [7]) but this version is sufficient for the problem 
at hand. 

Theorem A. 3 (Perron-Frobenius Theorem) : 

Let E be a finite set and let X n be a discrete time Markov chain on E with 
transition matrix Q. Up to merging some states, we suppose that X n admits 
an unique absorbing state {0}. We denote by Q* the matrix Q restricted to 
E* = E \ {0}. We suppose Q* to be irreducible. Then 
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1) There exists an unique quasi- stationary distribution (QSD) obtained as 
the only left eigenvector \i of Q* verifying 

Pi > ; y ] m = 1 

i 

The corresponding eigenvalue < p = e~ 8 < 1 is such that 

P M [T > n] = e- en 

2) The measure p is the Yaglom limit, i.e. 

G E* lim Pi[X n = j\T >n] = m 

n— >oo 

3) For every couple (i,j) of points in E* lim e 9n Pi[X n = j] = ■Kijij where 

n— >oo 

7T is the unique right eigenvector of Q* associated to p and such that 

n i > ! Hi Pi^i = 1 

4) For every couple of E* and every n,m G N* 

1 - Tn ^i[Tp > n + m] _ 7Tj e _ Sn 

n->oo Pj[To > n] TTj 

The result concerning the Yaglom limit can be refined 



Proposition A. 4 : 

We suppose Q to be C diagonalizable (this will be the case most of the time 
as the set of diagonalizable matrix of size n contains an open subset that is 
dense in Ai n (C)). We denote p = \\ > | Aa | > ... ^ |A n | the eigenvalues of 
Q. Let v be a measure on E, then : 

\\F v [X k €.\T >k]-»\\ 2 <cO^\ 

This result is an interpretation in our context of a classic result in numer- 
ical analysis on the power method, see e.g. [18]. 

For more information on the subject see [15] and [17]. 

A. 2. Stochastic approximation algorithms. 

This subsection has been inspired by [3] where you can find more information 
on the subject and complete proofs of the results. 

Here we will consider a family of discrete processes (X^ ) n eN taking values 
in K a compact subset of W 1 . The parameter N indexing the processes may 
take either real or integer values. We suppose that the X^ are defined on 
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a probability space (O, F, P) and denote by F^ the cr-algebra generated by 
{Xf ,i = l,...,n}. We suppose that 

where 

(i) F : R m — )• M m is a locally Lipschitz vector field 

(ii) E[U». 1 \F»] = 

(iii) There exists T ^ such that ||?7^|| ^ VT 

F will be called the mean field associated to X N . The following results 
may be proved under a broader set of hypotheses but these ones will be 
sufficient for our purposes. 

We denote by {(ft} the flow induced by F. So as to compare the trajectory 
of {(ft} with those of (X^) it's convenient to introduce the continuous in 
time process X N : R -4 M m defined by 

X N (k/N) = Xj* Vk e N 

and extended by linear interpolation on every [k/N, (k + 1)/N]. 
Let 

D N (T)=max r \\X N (t)-v t (X»)\\ 

be the variable measuring the distance between the trajectories t i— > X (t) 
and t H> (p t (X^). 

Theorem A. 5 : 

For every T > 0, there exists c > (depending only on F,T and T) such 
that, for every e > 0, and for N large enough : 

P[D N {T) > e] < 2me- £2cN 

See [3] for a proof. 

We now suppose that X N = (X^ jneN is a Markov chain defined on a 
countable set Kjy. We also suppose that X N admits at least one invariant 
probability measure 7Ttv. 

Theorem A. 6 : 

The limit set of {ttn} contains only probability measures that are invariant 
for the flow tpt ■ 
See [3] for a proof 

For more information on the subject see e.g. [3] and [2]. 
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